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DESCRIPTION 

Intelligent speech recognition with user interfaces 

The present invention relates to the field of automatic transformation of speech to text 

and especially-to-automatic text modifications of text which has been automatically 

5 converted from speech. The automatic text modification detects text portions according 
to modification rules, generates intelligent modification suggestions and interacts with a 
user having the final decision for the text modification. 

Speech recognition systems that transcribes speech to a written text are known in the 
prior art. Commercial speech recognition systems are nowadays widely distributed in 
10 the medical sector for example in hospitals and also in legal practices. Speech recog- 
nition for transcription of spoken speech to written text saves time and reduces costs 
since a transcription of a dictation has no longer to be performed by a typist. 

Typically a dictation not only contains text to be transcribed but also commands that 
have to be interpreted by the speech recognition system. Punctuation commands should 
15 not be literally transcribed as e.g. "colon" or "full stop". Punctuation commands, or 
formatting or highlighting commands should also be recognized and interpreted by an 
intelligent transcription system. The recognized text in combination with the interpreted 
commands finally yields a document which has to be corrected by a human proofreader 
or editor. 

20 Commercial speech recognition systems such as SpeechMagic™ of Philips Electronics 
N. V. and the ViaVoice ™ system of IBM Corporation feature text recognition as well as 
command interpretation. Both of these commercial speech recognition systems can be 
implemented into text processing software products for transcribing, editing, correcting 
and formatting text Furthermore, these commercial systems provide voice controlled 

25 interaction between a user and a personal computer. Interpreting voice commands 

activate menu options and other customized software functions as for example browsing 
the Internet. 
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Nevertheless a dictation inherently features ambiguous text portions such as e.g. 
numbers that have to be interpreted as a number or that have to be interpreted literally as 
a written word depending on the context of the spoken dictation. Such ambiguous text 
portions are easily misinterpreted by an automatic speech recognition system. 
5 Furthermore system-based interpretations o f text formatting or text highlighting 
commands might be erroneous. Such inevitably system-generated misinterpretations 
have to be manually corrected by a human proofreader which reduces the efficiency of 
the entire speech recognition system. A system-supported modification or correction of 
potentially ambiguous or misinterpreted text portions is therefore highly desirable in 
1 0 order to facilitate the proofreading. 

Specific text correction and text modification functions for text processing systems are 
known in the prior art WO 97/49043 describes a method and a system for verifying 
accuracy of spelling and grammatical composition of a document. In an electronic 
document a sentence is extracted and the words of the extracted sentence are checked 

15 against misspelling. When the system detects a misspelled word, an indication is 

displayed in a combined spelling and grammar dialogue box. The word as well as the 
entire sentence in which the spelling error occurred is displayed. Furthermore a spell 
checker program module receives suggestions being displayed in a suggestion list box 
within the combined spelling and grammar dialogue box. A user then inputs a command 

20 by selecting one of the command buttons of the combined spelling and grammar 
dialogue box. In response to the user selecting one of these command buttons the 
method performs the appropriate steps. In a similar way the method is applicable to 
grammar checking of sentences. 

US Pat. Nr. 6047300 describes a system and method for automatically correcting a 
25 misspelled word. In this system a correctly spelled alternate word is generated if a word 
is detected as a misspelled word. The misspelled word and the correctly spelled 
alternate word are compared according to a set of different criteria. If the results of 
various different criteria comparisons satisfy a selection criteria, then the misspelled 



-3- 



PHDE030359 EPP 



word is replaced by a correctly spelled alternate word Even though a word is detected 
as a misspelled word, the user may have intended that the word appears as entered To 
maintain the word as entered, an automatic replacement of the misspelled word must be 
~~5 overridden. In order to overiide~the replacement, lliellocument discloses a spelling 
embodiment including an exception list of exception words. An exception word has to 
be defined by the user and is not subject to replacement. The user may edit the 
exception list to add and to remove exception words. 

US Pat. Nr. 6047300 also discloses a spelling embodiment according to which the user 
10 may or may not receive notice when a misspelled word is replaced by a correctly spelt 
word If the user receives a replacement notice, then the user is aware of the 
replacement and may confirm or reject the replacement 

The above cited documents only refer to listings of spelling or improper grammatical 
compositions within electronic text documents. Ambiguous text portions that may arise 

1 5 from a speech to text transcription cannot be identified by the above mentioned methods 
because the ambiguous text portions are correctly spelt In the same way text formatting 
or text highlighting commands included in a dictation and being literally transcribed 
from an automatic speech recognition system are typically not detectable by means of 
the above mentioned correction and verifying systems. Generally, these systems are not 

20 adaptable for performing a context-based modification of an electronic text 

The present invention aims to provide a method, a system, a graphical user interface and 
a computer program product for an automatic text modification with user interaction of 
an electronic text generated by a speech to text recognizing system. 

25 The present invention provides an automatic text modification with user interactions. 
Preferably reliable modifying actions, such as e.g. the straightforward interpretations of 
non-ambiguous commands or non-ambiguous text portions are directly executed When 
in contrast non-reliable actions, such as e.g. ambiguous text portions or unresolvable 
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commands are detected, the method requests for human expertise prior to the execution 
of a modifying action. An executed modifying action as well as a request for human 
expertise is indicated to the user. In this way, the user gains an easy and effective access 
to modified text portions and/or potentially misinterpreted spoken commands and/or 
5 ambiguous text portions as well as other potential problems associated with a speech to 
text recognition. 

For example any kind of number is associated with an ambiguous text portion. Since a 
number can be interpreted as a number (which has to be written in digits) or as an 
enumeration or literally as a word, the speech to text recognition system requests for 
10 human expertise. A decision whether a number has to be written in digits, as an 
enumeration, or as a word is context dependent. Such ambiguous text portions are 
recognized automatically by the system and highlighted in the generated text. In this 
way the system gives intelligent hints to the proofreader about potential misinter- 
pretations that may have occurred in the speech to text transformation step. 

1 5 Not only numbers but also certain phrases or words can be subject to misinterpretation. 
The word "colon" for example may be written as "colon" (e.g. in medical reports) or as 
":" as typographical sign depending on the context. 

According to a preferred embodiment of the invention, the system features several rules 
to identify text portions within the recognized text that might be subject to a modifica- 
20 Hon. The generated text is displayed on a user interface for proofreading purposes. In 
order to fecilitate the proofreading, potential text modifications are highlighted within 
the text. Highlighting can be performed by any means of accentuation as e.g. different 
colour, different size, different font or different typeface of the text to be modified 

According to a further preferred embodiment of the invention, text portions being 
25 matched by at least one of the rules are automatically modified by the system and 

highligjited in the text In this way the proofreader can immediately identify those text 
portions that have been modified by the system. Furthermore the system provides an 
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undo function enabling the proofreader to correct automatically performed 
modifications of the text 

According to a further preferred embodiment of the invention, the rules provide a 
confidence value indicating a likelihood whether a matched text portion is subject to 

~5 modification? A texHnodificatibn is automatically p^drmed when the confidence value 
is above a first predefined threshold In this case the text modification is performed 
without making any annotation or any further suggestion. When the confidence value is 
below the first threshold but above a second threshold, the automatic modification is 
performed associated with an indication for the user and associated with appropriate 

10 undo information enabling the user to cancel the performed modification. When the 
confidence value is below the second threshold, a modification is not performed 
automatically but a suggestion is indicated to the user and the system requests for a 
decision to be made by the user whether the matched text portion has to be modified or 
not Typically the threshold values for the confidence value can be adapted to the 

1 5 proofreader's or user's preference. 

According to a further preferred embodiment of the invention, the text portions matched 
by the rules are not automatically modified by the system. Instead the proofreader's, or 
the user's expertise is requested in order to make a decision whether a modification 
should be performed or not Text portions matched by the rules are therefore highlighted 
20 in the text. The highlighted text portions can then easily be discovered by the proof- 
reader. The highlighted text is typically associated with one or several suggestions for 
the text modification. Typically the user has a possibility to accept or to reject the 
suggestions generated by the system. The text modification is finally performed in 
response to the user's decision. 

25 Depending on the type of text document, different context based rule modules can be 
applied in order to detect ambiguous or problematic text portions. The context based 
rule modules are for example specified for a legal practice, or a medical report 
Depending on the context, the rules not only detect ambiguous text portions but also 
refer to some unclear commands contained in the dictation. 
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Furthermore, commands such as "quote unquote" may be interpreted as a quoting of the 
next word only or as me beginning of a quoted region of unknown length. In such cases 
suggestions or hints are generated and highlighted in the text. The single rules may also 
be specified to detect inconsistencies in documents containing enumeration symbols 

5 such as "1, 2, 3,..." or "a), b), c),. . ." Si nce sp eakers are often not consistent in dictating 
all enumeration symbols the rules are designed for detecting missing items in a series of 
enumerations. In this case a hint or a suggestion is generated for the proofreader. 
Furthermore references to other text sections such as "the same" or "as above" may be 
transcribed literally or it may be common to resolve these references and to insert the 

10 corresponding text. Since any kind of apparatus has normally no chance to resolve such 
references, the system here provides some hint to the human proofreader if certain 
reference terms or phrases are detected. 

According to a further preferred embodiment of the invention, a suggestion is always 
generated and the appropriate text portion is always highlighted when two or more 
1 5 conflicting suggestions are provided for a text modification related to a distinct text 

portion. In such cases where at least two different rules provide different suggestions for 
a distinct text portion, the human expertise is definitely required. According to the 
confidence values of each conflicting suggestion the method provides a ranking or a list 
of suggestions from which the user or proofreader can make a selection. 

20 According to a further preferred embodiment of the invention, an automatic text 
modification is only performed when the automatic text modification comprises a 
number of editing operations which is below a predefined threshold value. When the 
number of text editing operations according to a distinct rule exceeds a distinct 
threshold value, the appropriate text modification is not performed as long as the 

25 proofreader has not made a decision. In this way the method asks for human expertise 
before it performs a large number of automatic editing operations. Therefore the number 
of potential undo operations to be performed by the proofreader is reduced to a 
minimum Such an interaction with the user saves time and costs. 
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According to a further preferred embodiment of the invention, the recognized text as 
well as the generated suggestions according to the different correction rules are 
outputted to a graphical user interface. The graphical user interface is designed to 
display the recognized text as well as to display the suggestions for potential text 
5 modification op eration s, A suggestio n can be displayed in a manifold of different ways. 
For example the suggestion can appear in the form of a suggestion menu which is 
positioned directly next to the highlighted text portion to which the suggestion relates. 
According to another embodiment of the invention, the different suggestions may 
appear in a separate window within the graphical user interface. 

10 According to a further preferred embodiment of the invention, a plurality of suggestions 
for various text portions are only displayed in response to the user's request Otherwise 
the graphical user interface may be overcrowded by a plurality of suggestions or 
suggestion lists. A user's request can be adapted in a manifold of different ways, as e.g. 
by clicking on a mouse button, shifting a mouse pointer on a highlighted text portion, 

1 5 touching with a finger on the appropriate position on the graphical user interface or 
simply via entering a universal key shortcut on a keyboard connected to the system. 

The appearance of various suggestions for a single highlighted text portion can also be 
adapted in a manifold of different ways. The single suggestions can appear according to 
a specified order (e.g. sorted by confidence value) as entries of a menu or as entries of a 
20 list, as well as in a completely disordered way. The way of appearance of the 
suggestions may be further specified by the user. 

According to a further preferred embodiment of the invention, a decision requested 
from the user can be performed in different ways. The user can either select one of the 
suggestions that have to be performed by the system or the user may manually enter an 
25 alternative suggestion to be performed by the system. The selection of a distinct 

suggestion can for example be realized with the help of a mouse pointer and a mouse 
click or with a universal keyboard shortcut Here any other type of interaction between 
the user and the graphical user interface is possible. 
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According to a further preferred embodiment of the invention, the selection of a distinct 
suggestion triggers associated side effects. When the system for example detects a 
missing enumeration, it suggests to implement this enumeration. When the user in turn 
decides to insert the missing enumeration, the system automatically gives a hint that a 
5 following letter might become subject to capitalization^tWs way the execution of 
some automatic modification according to a first rule invokes a second potential 
modification according to another rule. The user may further decide about the triggering 
of such side effects locally or globally in the document. 

The triggering of side effects due to a performed modification can further be controlled 
10 by means of a previously described confidence value associated with threshold values. 
In this way a distinction can be made, whether a side effect is automatically performed 
with or without indication to the user, or whether a side effect is automatically 
performed without any further interaction with the user. 



15 In the following, preferred embodiments of the invention will be described in greater 
detail by making reference to the drawings in which: 

Figure 1 is illustrative of a flow chart for performing a method of the invention, 
Figure 2 illustrates a flow chart for performing a second method of the invention, 
Figure 3 shows a block diagram of a preferred embodiment of the invention, 
20 Figure 4 shows a block diagram of a graphical user interface, 

Figure 5 is illustrative of a flow chart for triggering a modification rule. 

Figure 1 illustrates a flow chart for performing the method according to the invention. 
In the first step 100 speech is transformed to text In step 102 it is checked, which text 
25 regions are matched by one or several modification or inconsistency rules. In step 1 04 
problematic text regions are detected by means of conflicting applicable modification 
rules or by a match of some inconsistency rule. In step 106 the identified and detected 
text portions are highlighted within the text In step 108 the method creates several 
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suggestions for each highlighted text portion and provides a list of suggestions. In step 
1 10 the created list of suggestions is displayed on the graphical user interface if 
requested by the user. In step 1 12 the user selects one of the suggestions or the user 
manually inserts a text modification which is then inserted in the text 

5 _ _. 

Figure 2 illustrates a flow chart of a method of the invention in which automatic text 
modifications are performed. Similar as described in figure 1 in step 200 the speech is 
transformed to text In the next step 202 it is checked which regions of the recognized 
text are matched by one or several modification or inconsistency rules. According to the 

10 various rules, text portions potentially being subject to modification are detected by the 
method in step 204. In step 206 the method automatically performs text modifications 
according to the rules. Since these automatic text modifications can be erroneous they 
are higfilighted in the text in step 208 and provided with some undo information for the 
user, hi this way the method performs an automatic text modification but also indicates 

15 to the user that an automatic, hence potentially erroneous, modification has been 
performed in the text. 

Ideally, the method also provides a specific undo function such that the user can easily 
revoke text modifications performed by the automatic text modification system. 

Figure 3 shows a block diagram of a preferred embodiment of the invention based on a 
20 speech to text transformation system 302. Natural speech 300 is entered into the speech 
to text transfonnation system 302. The speech to text transformation system 302 
interacts with a user 304 and generates modified text 316. The speech to text 
transformation system 302 comprises a speech to text transformation module 306, a rule 
match detector module 308, a rule execution module 309 as well as a graphical user 
25 interfece 3 10. The speech to text transformation system 302 further comprises context 
based rule modules 312, 314. Each of the context based rule modules 312, 314 
comprises a database 318, 324, a first rule 320, 326, a second rule 322, 328 as well as 
additional rules not further specified here. 
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Incoming speech 300 is processed in the speech to text transformation module 306 
providing a recognized text. The rule match detector module 308 then applies one or 
several of the context based rule modules 312, 314 to the recognized text. The databases 
318, 324 as well as 1he single rules 320, 322, 326, 328 are specified for a distinct text 
5 scok The databases 318, 324 are for example specified forlegal practice or medical 
reports. In "a similar way the rules 320, 322, 326, 328 are specified to different fields of 
application. Based on the chosen context based rule module 312, 314, the rule match 
detector module 308 detects text portions within the recognized text mat might become 
subject to modification. 

1 0 Modifications of the detected text portions are performed by the rule execution module 

309. According to the user's preferences, an automatic modification may be directly 
performed by the rule execution module 309 or may be performed according to auser's 
decision. Depending on the predefined threshold and confidence values, a performed 
modification may be indicated to the user associated with undo information or not A 

15 requirement of a user decision is indicated to the user via the graphical user interface 

3 10. The interaction between the speech to text transformation system 302 with the user 
304 is handled via the graphical user interface 310. When the system has performed an 
automatic text modification the appropriate text portion is higblighted on the graphical 
user interface 310. Text portions whose modification requires a user's decision are also 

20 highlighted on the graphical user interlace 310. When the system generates suggestions 
for an automatic modification according to the rules 320, 322, 326, 328, the suggestions 
are also displayed via the graphical user interface 310. Execution of the user's decisions 
as well as the automatic text modifications into the recognized text finally give the 
modified text 316 which is outputted from the speech to text transformation system 302. 

25 Furthermore, when a text portion matches an inconsistency rule, applying to e.g. a 
missing enumeration, an unresolvable reference or other inconsistencies, a warn icon 
indicating a text inconsistency is generated on the graphical user interface 310. 

Figure 4 shows a block diagram of a graphical user interface 400 of the present 
invention. The graphical user interface 400 comprises a text window 402 as well as a 
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suggestion window 404. The text window 402 typically contains several highlighted 
text portions 406 indicating a potential modification or a warn icon of a text 
inconsistency. The highhghting of the text can be performed in different ways, such as 
e.g. different color, different font or other preferably visual indications. Various 
5 suggestions for the modific ation o f a highlighted text portion can be displayed bymeans_ 
of a suggestion list 41 0 appearing within the text window 402 or within the suggestion 
window 404. The suggestion window 404 as well as any list of suggestions 410, 412 
may be always present inside the graphical user interface 400 but may also only appear 
on a user's demand. 

10 With the help of a mouse pointer 408, the user can select certain highlighted text 
portions 406 for which the appropriate suggestion list 410, 412 or the suggestion 
window 404 appears. The selection of highlighted text portions 406 for which lists of 
suggestions 410, 412 appear can also be performed with the help of any other type of 
input means, such as e.g. a keyboard shortcut, a touch screen or even a speech command 

15 of the user. With the help of the same means, the user finally selects one of the provided 
suggestions of the suggestions lists 410, 412 or the user may manually enter an 
alternative text portion. 

Figure 5 is illustrative of a flow chart representing the execution of text modifications 
with respect to triggering of rules as side effects of performed text modifications. In a 

20 first step 500 it is checked which text portions of the recognized text are matched by 
one or several modification or inconsistency rules. In step 502, N text portions 
potentially being subject to an automatic text modification are detected and an index j is 
initiated (j=l). Step 504 compares the index j and Ihe number N of text portions being 
potentially subject to modification. If j is larger than N, the method proceeds with step 

25 518 and the modification ends. If in step 504 j is less or equal than N, then in step 506 
the first text portion (j=l) is highlighted within the recognized text. In step 508 the 
method provides a list of suggestions for the text modification which is displayed on the 
graphical user interface. In step 510 the interaction with the user is performed. Next the 
text portion j is modified according to the user interaction in step 512. 
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The following step 514 checks whether the performed modification of the text triggers any 
other of the text modification rules. For example when the first modification enters a 
missing punctuation such as a the proceeding word of the next sentence has to be 
capitalized accordingTo an other rule. When in step 514 the performed modification 
triggers such an other rule, the other rule is applied to the text portion in step 516. After the 
other rule has been applied to 1he designated text portion, the method returns to step 506 
and performs the same suggestion and interaction procedure for the selectedrule. When in 
contrast in step 514 no other rule is triggered by the performed modification, the indexj is 
increased by 1 and the method returns to step 504. 
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CLAIMS 



1. A method for speech to text transformation comprising the steps of: 

performing a speech recognition step to provide a text, 
applying at least one rule to the text for performing at least one automatic 
text modification, 
5 - outputting the text on a user interface, 

outputting at least one suggestion for the automatic text modification on 
the user interface. 

2. The method according to claim 1, wherein the at least one text modification is 
1 0 performed automatically and is highlighted in the text for a user's review. 

3. The method according to claim 1, wherein a portion of the text to which the at least 
one suggestion relates is highlighted in the text, and the at least one text modification is 
performed in response to a user's acceptance of the suggestion. 

15 

4. The method according to any one of the claims 1 to 3, the at least one rule providing a 
confidence value for the at least one modification, wherein the suggestion is only 
outputted for a user's review when the confidence value is below a threshold. 

20 5. The method according to any one of the claims 1 to 4, wherein suggestions are 

outputted for a user's selection when at least first and second conflicting suggestions for 
the text modification are provided. 
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6, The method according to any one of the claims 1 to 5, wherein the text modification 
comprises a number of editing operations, a suggestion for the text modification being 
outputted when the number of editing operations exceeds a predefined threshold value. 

5 7. The method according to any one"of the claims 1 to 6, wherein thetext and the 
suggestions are outputted to a graphical user interface. 

8. The method according to any one of the claims 1 to 7, wherein the at least one 
suggestion is displayed when the user enters a suggestion command 

10 

9. The method according to any one of the claims 2 to 8, wherein the user's review 
comprises selecting one of the suggestions by entering a selection command or 
manually entering a text modification. 

15 10. The method according to any one of the claims 1 to 9, wherein a first automatic text 
modification is performed according to at least one suggestion provided by a first rule, 
and further comprising applying of at least a second rule to the first text modification 
and outputting of at least a second suggestion for a second automatic text modification. 

20 1 1 . A speech to text transformation system (302) comprising: 

means for performing a speech recognition step to provide a text, 
means for applying at least one rule (320, 322, 326, 328) to the text for 
performing at least one automatic text modification, 
means for outputting the text on a user interface and outputting at least 

25 one suggestion for the automatic text modification. 

12. The system according to claim 1 1, comprising means for highlighting a text 
modification or a text portion to which relates at least one suggestion for a user's 
review. 
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13. The system according to claims 1 1 or 12, comprising means for generating at least 
one suggestion output when at least first and second conflicting suggestions for the text 
modification are provided or when the text modification comprises a number of editing 

5 operations, the number exceeding a predefined threshold value. 

14. The system according to any one of the claims 1 1 to 1 3, comprising means for 
displaying the at least one suggestion on a graphical user interface (310; 400) when the 
user (304) enters a suggestion command. 

10 

15. The system according to any one of the claims 1 1 to 14, comprising means for 
performing a first automatic text modification in accordance with a first suggestion 
provided by a first rule (320,...;326,...), and further comprising means for applying of at 
least a second rule (322,...;328,...) to the first text modification and outputting of at least 

15 a second suggestion for a second automatic text modification. 



16. A graphical user interface (310; 400) for a speech to text transformation system 
(302) having at least one rule (320, 322, 326, 328) for performing at least one automatic 
text modification and having at least one suggestion for the automatic text modification, 
20 the graphical user interfece (3 1 0; 400) comprising: 

means for generating a display of the text, 

means for highlighting text and highlighting text portions to which the at 
least one suggestion relates, 

means for displaying the at least one suggestion for the automatic text 
25 modification. 
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17. The graphical user interface according to claim 1 6, farther comprising means for 
entering a suggestion command by the user in order to initiate the display of the at least 
one suggestion. 

5 — 18. A computer program product for a speech to text transformation system comprising— 
program means for: 

performing a speech recognition step to provide a text, 

applying at least one rule to the text for performing at least one automatic 

text modification, 
10 - outputting the text on a user interface, 

outputting at least one suggestion for the automatic text modification for 

a user's review. 

19. The computer program product according to claim 1 8, the program means being 
1 5 adapted to perform the steps of: 

outputting suggestions for a user's selection when at least first and 
second conflicting suggestions for the text modification are provided, 
outputting a suggestion for a user's selection when the text modification 
comprises a number of editing operations exceeding a predefined 
20 threshold number. 

20. The computer program product according to claims 18 or 19, the program means 
being adapted to perform a first automatic text modification in accordance with a first 
suggestion provided by a first rule, and further comprising applying of at least a second 

25 rule to the first text modification and outputting of at least a second suggestion for a 
second automatic text modification 
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ABSTRACT 

Intelligent speech recognition with user interfaces 

The invention relates to a method, an apparatus, a Graphical User Interface and a 
computer program product for transforming speech to text. In a speech recognition step 

J5 a te xt is pr ovided to which an automatictext modification b ased on text modification 
rules is applied. Text portions to which the modification applies, are highlighted in the 
text and at least one suggestion for the modification is displayed to a user, being 
enabled to select one of the suggestions and therefore having control over the automatic 
text modification process. For example the rules identify text portions relating to 

1 0 ambiguous speech or missing text formatting characters that might have been 
misinterpreted in the speech recognition step. 
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